Test suites assess natural language processing models' performance on specific functionalities: cases of interest involving model robustness, fairness, or particular linguistic capabilities. They enable fine-grained evaluations of model aspects that would otherwise go unnoticed in standard evaluation datasets, but they do not address the problem of how to fix the failure cases. Previous work has explored functionality learning by fine-tuning models on suite data. While this improves performance on seen functionalities, it often does not generalize to unseen ones and can harm general performance. This paper analyses a fine-tuning-free approach to functionality learning. For each functionality in a suite, we generate a specification instruction that encodes it. We combine the obtained specification instructions to create specification-augmented prompts, which we feed to language models pre-trained on natural instruction data to generate suite predictions. A core aspect of our analysis is to measure the effect that including a set of specifications has on a held-out set of unseen, qualitatively different specifications. Our experiments across four tasks and models ranging from 80M to 175B parameters show that smaller models struggle to follow specification instructions. However, larger models (> 3B params.) can benefit from specifications and even generalize desirable behaviors across functionalities.

You might think your TV is'Super Bowl ready,' but are you really prepared for the big day? Here are some tips from columnist Marc Saltzman to make sure your TV is set up perfectly. For many football fans, getting a little "action" on the Super Bowl is as commonplace as the wings and beer. Whether it is betting on the outcome of the game itself or participating in "prop bets" (such as whether the opening coin toss will be heads or tails) or "boxes" (also known as "squares," where you try to predict the correct last digit of the NFC and AFC team's score), wagers of various kinds can often be found at Super Bowl parties across the country. That's in addition to the $138.5 million legally bet through licensed sports books in Nevada.